[data.search] Simplify poll logic and improve types #82545

lukasolson · 2020-11-03T23:47:51Z

Summary

This PR simplifies the async search polling observable and search strategy authoring.

Any new search strategy should be implementing a method that fetches partial data and formats it into a IKibanaSearchResponse.
pollSearch will periodically call this method, until response is complete or until an error or an abort have occurred.

For example:

export const customSearchStrategyProvider = (
  logger: Logger
): ISearchStrategy<CustomSearchStrategyRequest, CustomSearchStrategyResponse> => {
  return {
    search: ({ id, ...request }, options: IAsyncSearchOptions, { esClient, uiSettingsClient }) => {
      const search = async () => {
        const promise = id
          ? chekupOnSearch(...)
          : submitSearch(...);
        const response = await promise;
        return toCustomKibanaSearchResponse(response);
      };

      return pollSearch(search, options).pipe(tap((response) => (id = response.id)));
    },
  };
};

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

Co-authored-by: Anton Dosov <dosantappdev@gmail.com>

…search-as-scoped

lizozom · 2020-11-04T13:32:30Z

@elasticmachine merge upstream

lizozom · 2020-11-21T19:56:57Z

@elasticmachine merge upstream

This reverts commit 01e8a06.

lizozom · 2020-11-24T12:13:24Z

x-pack/plugins/security_solution/server/search_strategy/security_solution/index.ts

-      return es
-        .search({ ...request, params: dsl }, options, deps)
-        .pipe(mergeMap((esSearchRes) => queryFactory.parse(request, esSearchRes)));
+      return es.search({ ...request, params: dsl }, options, deps).pipe(


We now shim totals in the responses in the route handler.
This means that internally, all strategies should work with the new format {value: 123, relation: 'eq'}.
We shimmed the types here to avoid test failures.

patrykkopycinski

SIEM/Endpoint LGTM

kibanamachine · 2020-11-24T22:35:26Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 1c14a36

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`data`	603	599	-4
`dataEnhanced`	39	38	-1
total			-5

Distributable file count

id	before	after	diff
`default`	43051	43048	-3
`oss`	22580	22576	-4

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	977.4KB	973.5KB	-3.9KB
`dataEnhanced`	36.3KB	35.2KB	-1.1KB
total			-5.0KB

History

💚 Build #90013 succeeded 168214b
💔 Build #89926 failed d4ef755
💔 Build #89908 failed 2ba358f
💔 Build #89478 failed 98c2c47
💔 Build #89471 failed 7874b18

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* [Search] Add request context and asScoped pattern * Update docs * Unify interface for getting search client * Update examples/search_examples/server/my_strategy.ts Co-authored-by: Anton Dosov <dosantappdev@gmail.com> * Review feedback * Fix checks * Fix CI * Fix security search * Fix test * Fix test for reals * Fix types * [data.search] Refactor search polling and improve types * Fix & update tests & types * eql totals * doc * Revert "eql totals" This reverts commit 01e8a06. * lint * response type * shim inside strategies * shim for security * fix eql params Co-authored-by: Anton Dosov <dosantappdev@gmail.com> Co-authored-by: Liza K <liza.katz@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/plugins/data/server/index.ts # src/plugins/data/server/server.api.md

…84292) * [data.search] Simplify poll logic and improve types (#82545) * [Search] Add request context and asScoped pattern * Update docs * Unify interface for getting search client * Update examples/search_examples/server/my_strategy.ts Co-authored-by: Anton Dosov <dosantappdev@gmail.com> * Review feedback * Fix checks * Fix CI * Fix security search * Fix test * Fix test for reals * Fix types * [data.search] Refactor search polling and improve types * Fix & update tests & types * eql totals * doc * Revert "eql totals" This reverts commit 01e8a06. * lint * response type * shim inside strategies * shim for security * fix eql params Co-authored-by: Anton Dosov <dosantappdev@gmail.com> Co-authored-by: Liza K <liza.katz@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # src/plugins/data/server/index.ts # src/plugins/data/server/server.api.md * doc Co-authored-by: Lukas Olson <olson.lukas@gmail.com>

lukasolson and others added 20 commits October 15, 2020 15:16

[Search] Add request context and asScoped pattern

26c47bd

Update docs

36b8fcb

Merge branch 'master' into search-as-scoped

8274226

Unify interface for getting search client

c7296f0

Update examples/search_examples/server/my_strategy.ts

9dadfb2

Co-authored-by: Anton Dosov <dosantappdev@gmail.com>

Merge branch 'master' into search-as-scoped

ab55bcb

Review feedback

2b3d2a0

Merge branch 'search-as-scoped' of github.com:lukasolson/kibana into …

7a048a0

…search-as-scoped

Merge branch 'master' into search-as-scoped

2d0da67

Merge branch 'master' into search-as-scoped

f7cb5a1

Fix checks

6c9aec5

Fix CI

1ae55ef

Fix security search

4634fb4

Merge branch 'master' into search-as-scoped

c35fb49

Merge branch 'master' into search-as-scoped

1567995

Fix test

5f7c9ab

Fix test for reals

f1b515d

Merge branch 'master' into search-as-scoped

b0860e7

Fix types

11517ee

[data.search] Refactor search polling and improve types

b1e06b0

lukasolson added Feature:Search Querying infrastructure in Kibana Project:AsyncSearch Background search, partial results, async search services. labels Nov 3, 2020

lukasolson requested a review from a team as a code owner November 3, 2020 23:47

lukasolson requested a review from a team November 3, 2020 23:47

lukasolson requested review from a team as code owners November 3, 2020 23:47

lukasolson self-assigned this Nov 3, 2020

lukasolson changed the title ~~[data.search] Simplify poll logic and improve types~~ [skip-ci] [data.search] Simplify poll logic and improve types Nov 3, 2020

lukasolson added the WIP Work in progress label Nov 3, 2020

lizozom added v7.11.0 v8.0.0 labels Nov 6, 2020

lizozom changed the title ~~[skip-ci] [data.search] Simplify poll logic and improve types~~ [data.search] Simplify poll logic and improve types Nov 10, 2020

lizozom added the release_note:skip Skip the PR/issue when compiling release notes label Nov 10, 2020

Merge branch 'master' of github.com:elastic/kibana into pr/82545

5f41a97

lukasolson mentioned this pull request Nov 16, 2020

[Search] Fixes EQL search strategy #83064

Merged

2 tasks

lukasolson added 2 commits November 19, 2020 15:51

Merge branch 'master' into search-poll

8e72dce

Fix & update tests & types

cf2b31d

kibanamachine and others added 2 commits November 21, 2020 14:57

Merge branch 'master' into search-poll

10a2f92

eql totals

01e8a06

lizozom removed the WIP Work in progress label Nov 22, 2020

Liza K added 8 commits November 22, 2020 15:13

doc

7874b18

Revert "eql totals"

07d6b72

This reverts commit 01e8a06.

Merge branch 'master' of github.com:elastic/kibana into pr/82545

9ee668b

lint

b61ebc3

response type

98c2c47

Merge branch 'master' of github.com:elastic/kibana into pr/82545

09eaaac

shim inside strategies

2ba358f

shim for security

1ad2f0b

lizozom reviewed Nov 24, 2020

View reviewed changes

Liza K added 3 commits November 24, 2020 14:23

Merge branch 'master' of github.com:elastic/kibana into pr/82545

d4ef755

fix eql params

168214b

Merge branch 'master' of github.com:elastic/kibana into pr/82545

1c14a36

patrykkopycinski approved these changes Nov 24, 2020

View reviewed changes

lizozom merged commit f80da6c into elastic:master Nov 24, 2020

lizozom mentioned this pull request Nov 24, 2020

[7.x] [data.search] Simplify poll logic and improve types (#82545) #84292

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[data.search] Simplify poll logic and improve types #82545

[data.search] Simplify poll logic and improve types #82545

lukasolson commented Nov 3, 2020 •

edited by lizozom

Loading

lizozom commented Nov 4, 2020

lizozom commented Nov 21, 2020

lizozom Nov 24, 2020

patrykkopycinski left a comment

kibanamachine commented Nov 24, 2020

[data.search] Simplify poll logic and improve types #82545

[data.search] Simplify poll logic and improve types #82545

Conversation

lukasolson commented Nov 3, 2020 • edited by lizozom Loading

Summary

Checklist

For maintainers

lizozom commented Nov 4, 2020

lizozom commented Nov 21, 2020

lizozom Nov 24, 2020

Choose a reason for hiding this comment

patrykkopycinski left a comment

Choose a reason for hiding this comment

kibanamachine commented Nov 24, 2020

💚 Build Succeeded

Metrics [docs]

Module Count

Distributable file count

Page load bundle

History

lukasolson commented Nov 3, 2020 •

edited by lizozom

Loading